                         Installation Notes
                  QLogic's ESX iSCSI Offload Driver

                         QLogic Corporation

              Copyright (c) 2015 QLogic Corporation
                         All rights reserved

Table of Contents
=================

  Introduction
  OS'es Supported
  Limitations
  Load and Run Necessary iSCSI Software Components
  BNX2I Driver Dependencies
  BNX2I Module Parameters
  vSphere Client Parameters Setting
  Unloading and Disabling Driver
  Binding and Unbinding iSCSI Offload Adapter
  Important information while working with Linux iET Targets
  Making connections to iSCSI Target
  Maximize iSCSI Offload Connections
  Caveats


Introduction
============

This file describes the bnx2i ESX 4.1/5.0 drivers for the QLogic 10
Gigabit Ethernet iSCSI Adapter 10/100/1000/2500/10G Mbps PCI/PCI-X/PCIE
CNIC Network Controller.
The bnx2i driver enables iSCSI offload on QLogic 1/10 Gigabit
Ethernet iSCSI Adapter family of devices.


OS'es Supported
================
	ESX5.0, ESX5.1, ESX5.5


Limitations
===========

Due to limitations in user level stack and ARP resolution module,
current offload solution does not connect to targets over routers.

Load and Run Necessary iSCSI Software Components
================================================
QLogic 1/10 Gigabit Ethernet iSCSI Offload software suite consists of 4 kernel modules,
bnx2i, cnic, bnx2 and bnx2x.

Required software components can be loaded either manually or through system
services.

Manual loading :
----------------
   # vmkload_mod bnx2i

NOTES: unloading bnx2i driver is not support in ESX environment

Disabling bnx2i driver:
-----------------------
   # esxcfg-module -d bnx2i


BNX2I Driver Dependencies
=========================

The driver uses library functions in the scsi_transport_iscsi, bnx2, bnx2x,
and cnic. It is required to load these library modules before attempting to
load the driver or unresolved symbol errors will appear. Also caution should
be taken to not load mis-matched drivers, this could lead to disabling iscsi
functionality as a result of HSI (Host Software Interface) differences.


BNX2I Module Parameters
========================

Following optional parameters can be supplied as a command line arguments
to 'vmkload_mod' command or set as persistent configuration for auto loading.
QLogic does not expect or recommend the end user to experiment unless they
are clear about the impact of these parameters values.

All module parameter set through 'esxcfg-module -s' will only be effective
after reboot.

----------------------------
error_mask1 and error_mask2:
----------------------------
Description: "Config FW iSCSI Error Mask #", use to configure
	certain iSCSI protocol violation to treated either as warning
	or fatal error. All fatal iSCSI protocol voilations will result
	in session recovery (ERL 0). These are bit masks

Defaults: All violation will be treated as errors.

CAUTION: Do not meddle with 'error_mask' if you are not sure about the
	consequences. These values are to be discussed with QLogic
	development team on case by case basic. This is just a mechanism
	to work around iSCSI implementation issues on the target side and
	without proper knowledge of iSCSI protocol details, user are advised
	not to experiment with these parameters


------------
en_tcp_dack:
------------
Description: "Enable TCP Delayed ACK", enables/disables TCP delayed ACK
	feature on offloaded iSCSI connections

Defaults: TCP delayed ACK is ENABLED

e.g.
	# vmkload_mod bnx2i en_tcp_dack=0
		or
	# esxcfg-module -s "en_tcp_dack=0" bnx2i


-------------
time_stamps :
-------------
Description: "Enable TCP TimeStamps", enables/disables TCP time stamp feature
	on offloaded iSCSI connections

Defaults: TCP time stamp option is DISABLED

e.g.
	# vmkload_mod bnx2i time_stamps=1
		or
	# esxcfg-module -s "time_stamps=1" bnx2i


------------
ooo_enable :
------------
Description: "Enable TCP out-of-order Rx Feature", enables/disables TCP OOO
	handling feature on offloaded iSCSI connections

Defaults: TCP OOO rx feature is ENABLED

e.g.
	# vmkload_mod bnx2i ooo_enable=1
		or
	# esxcfg-module -s "ooo_enable=1" bnx2i


--------
sq_size:
--------
Description: "Configure SQ size", used to choose send queue size for offloaded
	connections and SQ size determines maximum SCSI commands that can be
	queued. SQ size also has a bearing on the number of connections that can
	be offloaded, as QP size increases, number of connections supported will
	decrease. With default values, 5706/5708 can offload 28 connections.

Defaults: 128
Range: 32 to 128
Note: QLogic validation is limited to power of 2, e.g. 32, 64, 128


--------
rq_size:
--------
Description: "Configure RQ size", used to choose size of asynchronous buffer
	queue size per offloaded connections and RQ size is not required be
	greater than 16 as it is used to place iSCSI ASYNC/NOP/REJECT messages
	and SCSI sense data.

Defaults: 16
Range: 16 to 32
Note: QLogic validation is limited to power of 2, e.g. 16, 32


-------------
tcp_buf_size:
-------------
Description: "TCP send/receive buffer size", used to choose size TCP Window
	size for offloaded connections.

Defaults: 65536
Range: 8192 to 262144
Note: QLogic validation is limited to power of 2, e.g. 8192, 65536, 131072


----------------
event_coal_min :
----------------
Description: "Event Coalescing Minimum Commands", performance tuning parameter
	used to control the minimum rate of interrupt generation by the iscsi
	firmware

Defaults: 24
Valid Values: 16-32
Note: This parameter is meant for developers to tune the event coalescing for
	performance adjustment and not intended for end users.


----------------
event_coal_div :
----------------
Description: "Event Coalescing Divide Factor", performance tuning parameter
	used to moderate the rate of interrupt generation by the iscsi firmware

Defaults: 1
Valid Values: 1,2,4,8
Note: QLogic did find a single digit improvement in IOPS numbers on 1G chips
	But QLogic has decided to disable interrupt coalescing for
	5706/5708/5709 as our IOPS numbers are more than double the competition.
	However we believe this parameter makes more sense to 5771x (10G)


-------------------
cmd_cmpl_per_work :
-------------------
Description: "Number of CQE's processed per work", performance tuning parameter
	used to moderate the rate of completion processing in host software

Defaults: 24
Valid Values: 16 - 32
Notes: QLogic recommends end users to not change this parameter. This is intended
to be used by developers only.


-------------------
bnx2i_chip_cmd_max :
-------------------
Description: "Max IOs queued to chip", performance tuning parameter used to
	moderate the rate of commands queued to 1G chipsets.

Defaults: 24
Valid Values: 16 - 32
Notes: QLogic recommends end users to not change this parameter. This is intended
to be used by developers only.


-------------
en_hba_poll :
-------------
Description: "Enable HBA poll timer", debug parameter and QLogic does not
	expect end users to enable this

Defaults: 0
Valid Values: 0 or 1
Notes: QLogic recommends end users to not change this parameter. This is intended
to be used by developers only.


-------------------------------
bnx2i_nopout_when_cmds_active :
-------------------------------
Description: "iSCSI NOOP even when connection is not idle", send unsolicited 
	iSCSI NOPOUT only when the pipe is idle (no SCSI traffic)

Defaults: 1
Valid Values: 1 or 0

-------------------
bnx2i_debug_level :
-------------------
Description: "Bit mask to enable/disable debug logs", debugging parameter

Defaults: 0
Valid Values: bit mask, currently lower 8-bits are supported
	0x01 - Initialization related logs
	0x02 - Conn Setup related logs
	0x04 - iSCSI TMF related logs
	0x08 - iSCSI NOP related logs
	0x10 - bnx2i / cnic interface exchanges log
	0x20 - ITT cleanup related logs
	0x40 - iSCSI Connection event logs
	0x80 - Session Recovery related logs

--------------------
max_bnx2x_sessions :
--------------------
Description: "Maximum number of 10G sessions", maximum offloaded sessions on
	QLogic 10 Gigabit Ethernet iSCSI Adapter devices, e.g. 57710, 57711, 57712 and 578xx

Defaults: 128
Valid Values: 1 to 128

-------------------
max_bnx2_sessions :
-------------------
Description: "Maximum number of 1G sessions", maximum offloaded sessions on
	QLogic 1 Gigabit Ethernet iSCSI Adapter devices, e.g. 5706, 5708 and 5709

Defaults: 64
Valid Values: 1 to 64

------------------
bnx2i_esx_mtu_max:
------------------
Description: "Max MTU Supported for Offload Sessions" maximum MTU supported
	on 1G QLogic devices, e.g. 5706, 5708 and 5709.

Defaults: 9000
Valid Values:	1500 to 9000

------------------
bnx2i_max_sectors:
------------------
Description: "Maximum sectors supported by the driver" maximum sectors supported
	on 1G QLogic devices, e.g. 5706, 5708 and 5709.

Defaults: -1
Valid Values: 64 to 256

------------------
bnx2i_max_task_pgs:
------------------
Description: "Maximum pages allocated by the driver" maximum pages allocated for
	iscsi task per connection on 1G QLogic devices, e.g. 5706, 5708 and 5709.

Defaults: 2
Valid Values: 2 to 256


vSphere Client Parameters Setting
==============================

Driver behaviour can be affected by the parameters set in Vsphere Client.
Following are such parameters which dictates the driver functionality.

-------------
LoginTimeout:
-------------
Description: Time in seconds initiator will wait for the Login response to finish.

Defaults: 5
Valid Values: 1 to 60

Users are adviced to use timeout value as at least 10 seconds.

Impact:
Value less than 10 seconds sometimes casue login request timeout during continuous
login-logout test scenarios.

Path:
iSCSI Initiator Properties->Advanced

Unloading and Disabling Driver
==============================

Unloading the driver is not supported in ESX, however bnx2i can be disabled and
inhibited from loading during boot

	#esxcfg-module -d bnx2i


Binding and Unbinding iSCSI Offload Adapter:
============================================
1. List QLogic 1/10 Gigabit Ethernet iSCSI Adapter adapter binding

	# esxcli swiscsi vmnic list -d <vmhba#>
		or
	# esxcli swiscsi nic list -d <vmhba#>

2. Associate iSCSI Offload Adapter to NIC

	# esxcli swiscsi nic add --nic <vmk#> --adapter <vmhba#>
		or
	# esxcli swiscsi nic add -n <vmk#> -d <vmhba#>

3. List all bound / online QLogic Adapters
	# vmkiscsid -x "select * from ifaces;"

4. Delete all iSCSI Offload bindings
	# vmkiscsid -x "delete from ifaces;"

5. Delete a specific iSCSI Offload binding
	# esxcli swiscsi nic remove -n <port_name> -d <vmhba#>



Important information while working with Linux iET Targets:
===========================================================
In order for ESX 4.1 to distinguish LUNS from different iET targets user has
to add 'ScsiID' and 'ScsiSN' to target definition in /etc/ietd.conf

e.g.
Target iqn.2011-05.com.broadcom:lun0
Lun 0 Path=/dev/ram0,Type=fileio,ScsiID=BDSK-001,ScsiSN=0001

Also please note that on ESX 5.0 when defining SCSI serial numbers, please
use 4 characters. Using less then 4 characters will cause VMware ESX
to will think that there are multiple paths to that disk.


Making connections to iSCSI Targets:
====================================

Please refer to ESX4.1 documentation for comprehensive list of ''
commands. Here is a sample list of commands to discover targets and
create iscsi connections to target.


Add static entry:
-----------------

	# vmkiscsi-tool -S -a "<ipaddr> <targetname>" <vmhba#>


iSCSI target discovery using 'SendTargets':
-------------------------------------------

	# vmkiscsi-tool -D -a <ipaddr> <vmhba#>


Query Discovery Results:
------------------------
	# vmkiscsi-tool -D  <vmhba#>


Rescan Adapter:
---------------
	# esxcfg-rescan <vmhba#>


List all Discovered SCSI LUN:
-----------------------------
	# esxcfg-mpath -b


Maximize iSCSI Offload Connections:
===================================
With default driver parameter set which includes 128 outsanding commands, bnx2i
can offload a following number of connections,
	5706/5708	- 28
	5709		- 43
	5771x		- 128
	578xx		- 128
This is no hard limit, just a simple on chip resource allocation math.
bnx2i will be able to offload > 28 connections on 1G devices by reducing
the shared queue size which in turn limits the maximum outstanding tasks
on a connection. Refer to section "BNX2I Module Parameters" for further
details on sq_size and rq_size. Driver logs the following message to syslog
when maximum allowed connection offload limit is reached -
 "bnx2i: unable to allocate iSCSI context resources"


Caveats:
========

1. iSCSI Connection over router:
--------------------------------
Also due to limitations in user daemon stack, current offload solution does not
support iSCSI connections to target reachable over routers.


2. iSCSI and FCoE offload coexistence (For ESX 5.0 and greater)
-------------------------------------------------
Currently, there are no driver/firmware/hardware restrictions with
iSCSI and FCoE connections being offloaded at the same time.  But
please, note that having both offloads running at the same time
is not officially supported by VMware.  Currently, VMware only
officially supports either an offloaded iSCSI or an offloaded 
FCoE connection on the same device because of restrictions in
the driver certification test suite.  Please contact VMware for
more information on this limitation.

